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Abstract 

By using the q-Gaussian distribution derived by the maximum entropy method for spatially- 
correlated TV-unit nonextensive systems, we have calculated the generalized Fisher information 
matrix of ge n e m fo r (61,62,63) = (/A?>°9' s), where u q , a 2 q and s denote the mean, variance and 
degree of spatial correlation, respectively, for a given entropic index q. It has been shown from the 
Cramer- Rao theorem that (1) an accuracy of an unbiased estimate of \x q is improved (degraded) 
by a negative (positive) correlation s, (2) that of ai is worsen with increasing s, and (3) that of s 
is much improved for s ~ —1/(N — 1) or s ~ 1.0 though it is worst at s = (N — 2)/2(N — 1). Our 
calculation provides a clear insight to the long-standing controversy whether the spatial correlation 
is beneficial or detrimental to decoding in neuronal ensembles. We discuss also a calculation of the 
(/-Gaussian distribution, applying the superstatistics to the Langevin model subjected to spatially- 
correlated inputs. 

PACS numbers: 89.70.Cf, 05.70.-a, 05.10.Gg 



*hideohasegawa@goo.jp 



I. INTRODUCTION 



It is well known that the Fisher information plays an important role in statistical me- 
chanics and information theory (for review see l|). The Fisher information is a useful tool 
in evaluating an accuracy of information decoding, providing the lower bound for estima- 
tion errors of unbiased estimates in the Cramer- Rao theorem The Fisher information 
expresses the metric tensor in the Riemannian space spanned by the probability distribution 
functions (PDFs) in the information geometry Calculations of the Fisher information 



have been made for various systems such as neuronal ensembles 



3-Q- 



Neurons in ensem- 



bles communicate information, emitting short voltage pulses called spikes, which propagate 



through axons and dendrites to neurons in the next stage (for review see 18j- 22], related 
references therein). Main issues on the neuronal code are whether the information is encoded 
in the rate of firings (rate code) or in the firing times (temporal code), and whether the infor- 
mation is encoded in the activity of a single (or very few) neuron or that of a large number 
of neurons (population code). A recent success in brain-machine interface 23( suggests that 
the population code for the firing rate is employed in sensory and motor neurons, although 
it is still unclear what kinds of codes are adopted in higher-level cortical neurons. 

The theoretical study of the Fisher information has been performed for a discussion on 

nn 

the accuracy of decoding and the efficiency of information transmission p[-ll7|. Calculations 
of the Fisher information have been made mainly for uncorrelated (independent) systems 
because of a mathematical simplicity. It has been shown that in independent systems, the 
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111 ]. However, the 



Fisher information increases proportionally to the ensemble size 0, [s, 
correlation among constituent elements is inevitable in real systems. In neuronal ensembles, 
for example, statistical dependence among consisting neurons would be expected because 



each neuron may receive t 



erally interconnected 



re same external inputs and because consisting neurons are gen- 



22] . There has been a long-standing controversy how correlation 



affects the efficiency of population coding. Some researchers have shown that the correlation 
enhances the effectiveness of neural population code ^ 
the correlation hinders the population code |3, S, E, 



12 L while some have claimed that 
111 ] . In particular, the Fisher in- 
formation is shown to saturate to a finite value as the system size grows in the presence of a 
positive correlation 
decoding. 



111 ]. This raises questions on the role of correlation in information 
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In the last decade, much attention has been paid to the nonextensive statistics since 



Tsallis proposed the so-called Tsallis entropy S q . For iV-unit systems, S q is given by 
MML\ 

k B 



24 



S n 



q-1 



1 



p(x) q dx 



(1) 



where q is the entropic index, k B the Boltzmann constant, x = {xi} {i — 1 to N), dx = 
Yli=\ dxi, and p(x) denotes the multivariate PDF. In the limit of q — > 1.0, the Tsallis entropy 
given by Eq. ([T]) reduces to the Boltzmann-Gibbs-Shannon entropy, 

Si = —k B J p(x) \np(x) dx. (2) 

The Tsallis entropy is non-additive because for p(A U B) = p(A) p(B), we obtain 

S q (A U B) = S g (A) + S q (B) - ^^l Sq (A)S q (B). (3) 



The Tsallis entropy is super-extensive, extensive and sub-extensive for q < 1, g = 1 and 
g > 1, respectively, and q— 1 expresses the degree of the nonextensivity. The PDF of p(a;) in 
Eq. ([!]) is obtained by using the maximum entropy method (MEM) for the Tsallis ent ropy 



with some constraints. There are four possible M 



un-normalized method 



Ms at the moment: original method 24j, 
and the optimal Lagrange multiplier 



normalized method 

(OLM) method 29]. The four methods are equivalent in the sense that distributions derived 
in them are easily transformed each other 30J • A comparison among the four MEMs is made 
in Ref. [271 ] . The Tsallis entropy is a basis of the nonextensive statistics, which has been 
successfully applied to a wide class of systems with the long-range interaction and/or non- 



equilibrium (quasi-equilibrium) states [26l. 1271. l3ll|. 



One of alternative approaches to the nonextensive statistics besides the MEM is the 



superstatistics 



32 



33 



34j (for a recent review, see [35|]). In the superstatistics, it is assumed 



that locally the equilibrium state of a given system is described by the Boltzmann-Gibbs 



by 



statistics and its global properties may be expressed 
intensive parameter (i.e., the inverse temperature) 
adopted in many kinds of subjects such as hydrodynamic turbulence [36 
39] and solar flares [40] . 



a superposition over the fluctuating 



32|-[35|. The superstatistics has been 



37|, 



38], cosmic ray 



4l| 



re generalized Fisher information (GFI) in the nonextensive statistics is defined by 



48] 



99„6 



qE 



d\np(x)\ ( d\np[x 



d9 r . 



(4) 
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where E[-] stands for the expectation value over the PDF of p(x) [= p(x\0)], and 9 pa- 
rameters specifying the PDF. Equation (T4J) is derived from the gen eralized Kullback-Leibler 
divergence which is in conformity with the Tsallis entropy 41]- 48] . In the limit of q — > 1.0, 
the GFI given by Eq. (J3J) reduces to the conventional one. In a previous paper 49J, we 
discussed the effect of the spatial correlation on the Tsallis entropy and the GFI, calculating 
S g and gee for 9 = p q , where \i q stands for mean value [Eq. (jSJ)]. It is the purpose of the 
present paper to extend the calculation to the GFI matrix of ge n e m for 62, #3) = (p q , cr^, 
s), where a 2 q and s express variance and degree of the spatial correlation, respectively [Eqs. 
(J7j) and (jHJ)]. We will investigate the dependence of the GFI on s, N and q, by using the 



PDF derived by the OLM-MEM 



291 ] . Such detailed calculations of the GFI matrix have 



not been reported even for the extensive system (q = 1.0), as far as the author is aware 
of. The calculated GFI is expected to provide us with a clear insight to the controversy on 
a role of the spatial correlation discussed above. Quite recently, we have pointed out the 
possibility that input information to neuronal ensembles may be carried not only by mean 
but also by variance and/or correlation in firing rate within the population code hypothesis 



5(J 



511 ] . The inverse of the calculated GFI matrix expresses an accuracy of decoding when 



input information is carried by such population codes. 

The paper is organized as follows. In Sec. II, we obtain the PDF by the OLM-MEM for 
spatially-correlated nonextensive systems. In Sec. Ill, the maximum likelihood estimator for 
the inference of the parameters is discussed. In Sec. IV, analytic expressions for elements 
of the GFI matrix are presented with some model calculations. In Sec. V, the PDF for 



the 



jangevin model with spatially-correlated inputs is calculated within the superstatistics 



331 ] . which is compared to that derived by the MEM in Sec. II. Section VI is devoted 



;o conclusion with the relevance of our calculation to decoding in neuronal population code 
50, bj- 



II. MAXIMUM ENTROPY METHOD 



A. Probability distribution function 



We consider spatially-correlated iV-unit nonextensive systems, for which the Tsallis en- 



tropy is given by Eq. (HJ [2J, |25|. We derive the PDF, p(x), by using the OLM-MEM [29] 
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for the Tsallis entropy, imposing the constraints given by |49| 

1 

I'q 



p(x) dx, 

i 
i 



(5) 
(6) 

(7) 

(8) 



where E q [-] denotes an expectation value averaged over the escort distribution function of 

p(x) q 



(9) 



f p(x) q dx 

The OLM-MEM with the constraints given by Eqs. (|Sj)-(jHJ) leads to the PDF given by (for 



details, see Appendix B of Ref. 



1 



49|) 



P( x ) = ^ex P(? 

^q 



( 1 \ N N 

\ 9 q J i=1 7=1 



(10) 



with 



^l\ N/2 rriV 



9-1 

r s (27ra 2 Q ) N/2 



TTiv r> I 1 _L i_ 

lli=l D I 2> o-l 2 



2u a a?;\ N / 2 priV jj (l 1 1 (i+1) 
lli=l I 2' 1— g 2 



^"9 
1-9 



for 1 < q < 3, 
for 5 = 1, 
for q < 1, 



(11) 



Cl 



CO 5»j + Cl (1 - <5y), 

[i + (jv - m 

(i- s )[i + (N-i) s y 

s 

~ (i- s )[i + (N-i) s y 

{(l-8) N - 1 [l + (N-l)s]} 1 ' 2 , 
[{N + 2) - Nq] 



(12) 
(13) 

(14) 

(15) 
(16) 



where B(p,q) denotes the beta function and exp Jx) the g-exponential function defined by 



exp (x) = [1 + (1 - q)x 



1/(1-9) 



(17) 



with [x] + = max(i, 0). We hereafter assume that the entropic index q takes a value, 



< q < 1 + - < 3, 



(18) 



because p(x) given by Eq. (TTDT) has the probability properties with u q > for q < 1 + 2/N 



52]. 



and because the Tsallis entropy is stable for q > 

In the limit of q = 1.0, the PDF given by Eq. (flOl) becomes the multivariate Gaussian 
distribution given by 



P( x ) = TT ex P 



(2^2) Y^ C ij( x i ~ ^)( X i ~ ^l) 
\ 1 / ij 



(19) 



III. MAXIMUM LIKELIHOOD ESTIMATOR 

The logarithmic likelihood estimator for M sets of data of x m = {xi m } (i — 1 to N, 
m = 1 to M) is given by 



At 



m=l 



1 \ M 

lnL(0) = ^lnp(a: Jfe |0) = -(— ■ - j ^ In C/(a; m ) - M In Z, 

' m=l 



with 



U(x m ) 1 -)- % ^ Cjj (Xj m [^q)(%jin f^q) • 

2,Vq<?t ~ 



Variational conditions for parameters of 9 = fi q , a 2 and s lead to 



ainL 

dfig 

dlnL 
~daf 

dlnL 

ds 



1 M r 

-iEE— 



Cij{Xi m flq) 



1 <? m=l ij 
1 ^ 

' y m=l 
M 



Cij(x im - fi q )(x jm - fl q ) MN 



U(x r 



EE 



(dCij/ ds)(Xi m — flq){Xj m — fig) 



1 q 771=1 ij 



U(x m ) 



+ 



MN(N - 1) 



2(l-s)[l + (iV-l) s ] 



(20) 



(21) 



(22) 
(23) 



(24) 
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After some calculations using Eqs. (fT2l)-(ITlj). ([22]) -([21]), we obtain 

S m Y2i x imU(x m ) 



Vq 



a 



v q MN 



UN 



s cr 



v q MN(N - 1) 



U(x m ) ' 

m 1 



U(x r 



from which fi q , <r 2 and s are self-consistently determined. 
In the case of q = 1.0, Eqs. ([25]) - ([27]) become 

J^YY Xim ^ 



s a\ 



MN 



MN 



1 



MN(N - 1) 



Y Y Y ( Xim ~ Vl)(%jm - Hi). 



(25) 
(26) 

(27) 



(28) 
(29) 
(30) 



IV. GENERALIZED FISHER INFORMATION 



We have calculated elements of the GFI matrix given by Eq. ([4]) with a basis of 
(6*i, 9 2 ,0 3 ) = (fi q , <Jg,s), as given by (for details, see the Appendix) 



/ 



N 



\ 



a*[l + (N-l)s} 










Nu q 







N{N-l)v q s 



\ 



2am-s)[l + (N-l)s} 



N(N-l)u q s N(N + (N -l)u q s 2 } 



2(Tg(l — s)[l + (N — l)s] 2(l-s) 2 [l + (iV-l) S ] 2 / 



The positive definiteness of 
and q given by 



(31) 



in Eq. (1311) imposes the condition on conceivable values of s 



(N-l) 



sl < s < 1, 



^ 1 + iV' 



(32) 
(33) 
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The physical origin of Eq. (1321) is expressed by (see Appendix C in Ref. [491 ]) 

< E q [(X - N ) 2 ] < 1 £ E M X * - thf\ = a l ( 34 ) 



which signifies that the global fluctuation in X (= N~ l £\ x i) is smaller than the average 
of local fluctuations in The condition given by Eq. fl33|) is satisfied by q in Eq. ffT8]) . 

In the limit of q = 1.0 where v q = 1.0, Eq. ( l3TT) reduces to 

/ N \ 



af[l + (N-l) 8 ] 










N 

N(N - l)s 







N(N-l)s 



2af(l - s)[l + (N - l)s] 
N(N - 1)[1 + (N- l)s 2 } 



2o\{\ - s)[l + (N - l)s] 2(l-s) 2 [l + (iV-l)s] 2 / 



(35) 



which is in agreement with the result obtained directly from the multivariate Gaussian 
distribution given by Eq. ffTOl) . 

In the limit of s = 0.0 (i.e., no correlation), the GFI matrix given by Eq. fl3T|) becomes 



V 



N 

- 

^ 
2 ^ 4 











\ 



N(N- 1) 



(36) 



whose elements of and agree with those obtained previously in Ref. |48|. 

The Cramer-Rao theorem implies that the lower bound of an unbiased estimate of the 
parameters is expressed by the inverse of the GFI matrix, which is given by 



/ a 2 [l + (N - 1) 8 ] 



G" 1 



N 





2a A q [l + {N- l)u q s 2 } 2a 2 q s{l - s)[l + (N - l)s] 



\ 



N 



2a 2 q s(l - s)[l + (N - l)s] 2(1 - s) 2 [l + (N - l)sf 



N 



N(N — 1) 
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(37) 




FIG. 1: (Color online) The s dependence of inverses of the GFI, h IMq ^ tq (solid curves), 
(dashed curves) and h ss (chain curves), with (a) N = 2 and (b) N = 10 for various q (fig = 0.0 
and a\ = 1.0). 

Equations (I3T1) and ( |371) are the main result of our study. In what follows, we examine 
the s, N and q dependence of the inversed GFI matrix of h 9e = (G^ 1 )^ with some model 
calculations which are presented in Figs. [Q131 

The s dependence 

Equation (1371) shows (1) = 0.0 at s — sl, (2) h a 2 a 2 has a minimum at s — 0.0, and (3) 
h ss vanishes at s = sl and s = 1.0. The maximum of h ss locates at s = (N — 2)/2(N — 1) = 



9 




logio N 



FIG. 2: (Color online) The N dependence of inverses of the GFI, h^ q ^ q (solid curves), h a 2 a 2 
(dashed curves) and h ss (chain curves), for s = 0.0 and s = 0.5 (q = 1.0, \i q = 0.0 and a 2 = 1.0). 



sm, which becomes sm = 0.5 for a large N. Figure Ufa) shows the s dependence of the 
inverse of the GFI for N = 2 which is expressed by 

\ 



G" 1 



/ + 3) 

2 


V 











(7^(1+ V q S 2 ) 2 



a 2 Q s(l-s 2 ) (1-8 



2\2 



(38) 



/ 



With increasing s from s = Sl = —1.0, h^ qflq is linearly increased. h ss and h a 2 a 2 are 
symmetric with respect to s = 0.0 where h ss (h a 2 a 2) has a maximum (minimum). Figure 
1(b) shows a similar plot for iV = 10 for which sl = —0.11. With increasing s from 
s = —0.11, hp n is linearly increased. h ss has a maximum at s — sm = 0.44 and vanishes 
at s = —0.11 and s = 1.0. 



The N dependence 

We note in Eq. f|3T|) that for s — 0, the GFI is proportional to N. For a finite positive s, 
however, they show the saturation when N is increased: for iV — > oo, we obtain h llqfJLq = a 2 s, 
h a 2 a 2 = 2<jqS 2 and h ss = 2s 2 (l — s) 2 . For a negative s, inverse matrix elements tend to 
vanish as iV approaches (1 + |s|)/|s|. The calculated iV dependence of hoe is plotted in Fig. 
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FIG. 3: (Color online) The q dependence of inverses of the GFI, h a 2 a 2, for s = 0.0 (dashed curves) 
and s = 0.5 (solid curves) with N = 2 and N = 10 (/x g = 0.0 and a 2 q = 1.0). 

[21 where inversed matrix elements for s = 0.5 saturate at N > 10, although those for s = 0.0 
is proportional to N^ 1 . 

The g dependence 

Equation fl37j) shows that fyu 9Ata and /i ss are independent of q, while /ic^o-a j s increased 
with increasing q from g = 0. This increase is due to a factor of u q in Eq. ([IS]) , which is 
decreased with increasing g and which diverges at 1 + 2/iV: note that v q = N/2 + 1, 1.0 and 
0.0 for q = 0.0, q = 1.0 and q = 1 + 2/iV, respectively. The calculated g dependence of /i^o- 2 
is plotted in Fig. [31 where it diverges at q = 2.0 (q = 1.2) for N = 2 (N = 10). 



V. DISCUSSION 



We have discussed the GFI for the g-Gaussian distribution derived by the MEM 24. |25. 



291 ] . It is possible to derive the g-Gaussian distribution by using the Langevin model within 



the superstatistics 



32 



331 ] . We consider an ensemble consisting of iV elements in a given 
system. The dynamics of a variable Xi (i = 1 to N) is assumed to be described by the 



11 



Langevin model given by 



^ = -Ax, + !,(*). (39) 



Here A denotes the relaxation rate and input signals Ii(t) have variability defined by 

h(t) = I(t)+5Ii(t), (40) 

with 

(8Ii{t)) = 0, (41) 
(SliftSItf)) = 2D[5 y + sj(l - 6ij)]5(t - 0, (42) 

where the bracket (•) signifies the ensemble average, and 2D and sj denote the variance and 
degree of the spatial correlation, respectively. The variability in Eq. (j42p arises from noise 
and/or heterogeneity in consisting elements. The origin of the spatial correlation may be 
common external inputs and/or couplings among elements. 

The Fokker-Planck equation for the PDF of ir(x,t) for x = {x{\ is given by 

i 1 i j 1 3 

with the covariance matrix Q whose elements are given by 

Qij = 5 ij + s I {l-5 ij ). (44) 
The solution of the FPE (T431 is given by 

^ t)= { r s [2na 2 ] N / 2 ) ^ (~2^ 5 5 ^ ^ ~ ^ ~ ' (45) 
where fi, a 2 and s obey equations of motion given (argument t being suppressed) 

! = + (46) 
: -2Aa 2 + 2D, (47) 



dt 

ds (2D 
dt ~ ~\~o* 



[s-sj), (48) 



Cij and r s being defined by Eqs. (jl2|) and (115]) . respectively. We note in Eqs. fl46|) - fj48l that 
//(£) is decoupled from a 2 {t) and s(i), and that cr 2 (t) is independent of s(t) although s(t) 
depends on cr 2 (t). In the stationary state, we obtain 

/i = //A, a 2 = p s = sj. (49) 
12 



After the concept in the superstatistics |32|, |33j, [34j, |35[ , we assume that a model parameter 
of (3 (= 1/cr 2 = X/D) fluctuates, and that its distribution is expressed by the ^-distribution 
with rank n, 



f(P) 



re/2 



n 



re/2-1 -n/3/2/3o 



fn = l,2,--0 



(50) 



r(n/2) V2/V 

where is the gamma function. Average and variance of (3 are given by (P)p = (3q and 
{(P 2 )p - Pi) /Pi = 2/n, respectively. Taking the average of tt(x) over /(/3), we obtain the 
stationary distribution given by 



p(aj) 



71(X) f(P) dP, 



Y q exPq (~ 2^ S ~ ^ ~ ^ j 



(51) 
(52) 



with 



27o 



AT/2 



1 + 



//-I 

r s (2^)^ 
2 



nf =1 5(|,^T-f) forg>l, 

for 5 = 1, 



(53) 
(54) 

A)(iV + n) 2ft ' (55) 

where r s is given by Eq. (|T5|) . In the limit of n — > oo (9 — > 1.0) where — ► — ft), 
the PDF reduces to the multivariate Gaussian distribution given by 



7, 



(N + ri) 

n (N + 2) - Nq 



(56) 



which agrees with Eq. (I4"5l) for ft = A/£> = 1/ct 2 . 

We note that the PDF given by Eq. ( 1521) is equivalent to that given by Eq. ( fTUj) derived 
by the MEM when we read fi = fi q and 7 g = i^cx 2 , besides the fact that the former is defined 
for 1 < q < [1 + 2/{N + n)} < 2 [Eq. (JSU)] while the latter for < q < (1 + 2/iV) < 3 [Eq. 

In the limit of s = (i.e., no spatial correlation), Eq. ( l52l) reduces to 



p(a;) oc exp 9 (^-7^^2( x i ~ V) 2 ^j , 



(57) 
(58) 
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with 



p(xi) oc exp ? ( -j^-fa - /i) 2 ) , (59) 



where the g-product is defined by [53] 



x(g) q y = [x^ + y 1 ^- l]V(i-9). (60) 
Note that in deriving Eq. (I58I) . the normalization factors of p(xj) are not taken into account. 



VI. CONCLUSION 

We have calculated the GFI matrix in spatially-correlated nonextensive systems. From 
the Cramer-Rao theorem, the calculated GFI implies the followings: (i) an accuracy of an 
estimate of \i q is improved (degraded) by a negative (positive) correlation, (ii) that of a 2 q is 
worsen with increasing s, (iii) that of s is much improved for s ~ — 1/(N — 1) and s ~ 1.0 
while it is worst at s = sm = (N — 2)/2(N — 1), (iv) those of all parameters are improved 
with increasing N, and (v) that of a q is worsen with increasing q at q > 1 and its estimation 
is impossible for q > 1 + 2/N, while those of \i q and s are independent of q. 

The points (i) and (iv) are consistent with previous results for extensive systems (q = 1.0) 
5, S [l0]. The point (iii) shows that if input information is carried by synchrony within 



the population code hypothesis [50, [5J| , its decoding accuracy may be improved either by 
small 0I large correlation, raclepeadently of q [the porat (v)]. Our caleulation eoncems the 
long-standing controversy on a role of the synchrony in neuronal ensembles |5l|-|12|. 
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APPENDIX: CALCULATIONS OF THE GENERALIZED FISHER INFORMA- 
TION MATRIX 



First we express PDFs of p(x) in Eq. (imp in a compact form given by 

u- b 



with 



p(x) 



U = 1 + a 2 ^ Cjj(xj - n q ){xj - fig), 

» 3 



(A.l) 



(A.2) 



nil 5(1, 6 -|) for Kg<3, 



Z„ = < 



r s (27ra„ 2 ) 7V / 2 for g = 1, 



^ » for g < 1, 



(A.3) 



2^ 



1/2 



9-1 



By using the unitary transformation, Eq. flA.2[) is transformed to 



Vi 



(A.4) 
(A.5) 



(A.6) 



where \ and i/j express eigen-values and eigen-vactors, respectively. We obtain \ given by 

1 



Xi 



[l + (N-l)s] 
1 



l-s) 



for z = 1, 
for 1 < i < N. 



(A.7) 
(A.8) 



Explicit expressions for yt are not necessary for our discussion, except for yi given by 

Vi = -?Tf^2( x i- n)- ( a - 9 ) 



N . 

Taking the derivatives of \np(x) with respect to parameters of fi q , a 1 and s, and performing 
tedious calculations with Eq. (j4j), we may obtain the GFI matrix elements given by Eq. 
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(!3T|) . In deriving them, we have employed the following expectation values: 



E 



E 



E 




4a 4 6(6 + l)A 2 ' 



4a 4 b(b + l)AjA 



2a 2 b\i 



2a?b{b+l)\ i 
3 



{b - N/2) 



(b - N/2) 



1 



b 



1 



for % ^ j, 



(A.13) 



(A. 14) 



(A.10) 



(A.12) 



(A.ll) 



where E[-] denotes the average over p(x). 
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